Determining image-based product from digital image collection

ABSTRACT

A method of making an image-based product includes providing an image collection having a plurality of digital images, each digital image having an image type; providing one or more image-type distributions, each image-type distribution corresponding to a theme and including a distribution of image types related to the theme; using a processor to automatically compare the image types of the digital images in the image collection to the image types in the image-type distribution; using the processor to automatically determine a match between the image types in the image collection and the image types in the image-type distribution; and selecting a group of digital images from the image collection having a distribution of image types specified by the determined matching image-type distribution. The method further includes assembling the digital images in the selected group of images into an image-based product and causing the construction of the image-based product.

CROSS REFERENCE TO RELATED APPLICATIONS

Reference is made to commonly-assigned, U.S. patent application Ser. No.13/222,605, entitled “Automated Photo-Product Specification Method”,filed Aug. 31, 2011, by Ronald S. Cok, et al.; U.S. patent applicationSer. No. 13/222,650, entitled “Automated Photo-Product SpecificationMethod”, filed Aug. 31, 2011, by Ronald S. Cok et al.; U.S. patentapplication Ser. No. 13/222,699, entitled “Automated Photo-ProductSpecification Method”, filed Aug. 31, 2011, by Ronald S. Cok, et al.;U.S. patent application Ser. No. 13/222,799, entitled “AutomatedPhoto-Product Specification Method”, filed Aug. 31, 2011, by Ronald S.Cok, et al.; and U.S. patent application Ser. No. 13/278,287, entitled“Making Image-Based Product From Digital Image Collection”, filed Oct.21, 2011, the disclosures of which are incorporated herein.

FIELD OF THE INVENTION

The present invention relates to photographic products having digitalimages that include multiple digital images and more specifically toautomated methods for determining and matching image-type distributionsin an image collection and selecting digital images from the imagecollection to be included in an image-based product.

BACKGROUND OF THE INVENTION

Products that include images are a popular keepsake or gift for manypeople. Such products typically include an image captured by a digitalcamera that is inserted into the product and is intended to enhance theproduct, the presentation of the image, or to provide storage for theimage. Examples of such products include picture albums, photo-collages,posters, picture calendars, picture mugs, t-shirts and other textileproducts, picture ornaments, picture mouse pads, and picture post cards.Products such as picture albums, photo-collages, and picture calendarsinclude multiple images. Products that include multiple images aredesignated herein as image-based products.

When designing or specifying photographic products, it is desirable toselect a variety of images that provide interest, aesthetic appeal, andnarrative structure. For example, a selection of images having differentsubjects, taken at different times under different conditions that tella story can provide interest. In contrast, in a consumer product, aselection of similar images of the same subject taken under similarconditions is unlikely to be as interesting.

In conventional practice, images for a photographic product are selectedby a product designer or customer, either manually or with the help oftools. For example, graphic and imaging software tools are available toassist a user in laying out a multi-image product, such as a photo-book.Similarly, on-line tools available over the internet from a remotecomputer server enable users to specify photographic products. The KodakGallery provides such image-product tools. However, in many casesconsumers have a large number of images, for example, stored in anelectronic album in a computer-controlled electronic storage deviceusing desktop or on-line imaging software tools. The selection of anappropriate variety of images from the large number of images availablecan be tedious, difficult, and time consuming.

Imaging tools for automating the specification of photographic productsare known in the prior art. For example, tools for automating the layoutand ordering of images in a photo-book are available from the KodakGallery as are methods for automatically organizing images in acollection into groups of images representative of an event. It is alsoknown to divide groups of images representative of an event into smallergroups representative of sub-events within the context of a largerevent. For example, images can be segmented into event groups orsub-event groups based on the times at which the images in a collectionwere taken. U.S. Pat. No. 7,366,994, incorporated by reference herein inits entirety, describes organizing digital objects according to ahistogram timeline in which digital images can be grouped by time ofimage capture. U.S. Patent Application Publication No. 2007/0008321,incorporated by reference herein in its entirety, describes identifyingimages of special events based on time of image capture.

Semantic analyses of digital images are also known in the art. Forexample, U.S. Pat. No. 7,035,467, incorporated by reference herein inits entirety, describes a method for determining the general semantictheme of a group of images using a confidence measure derived fromfeature extraction. Scene content similarity between digital images canalso be used to indicate digital image membership in a group of digitalimages representative of an event. For example, images having similarcolor histograms can belong to the same event.

U.S. Patent Application Publication No. 2008/0304808, incorporated byreference herein in its entirety, describes a method and system forautomatically producing an image product based on media assets stored ina database. A number of stored digital media files are analyzed todetermine their semantic relationship to an event and are classifiedaccording to requirements and semantic rules for generating an imageproduct. Rule sets are applied to assets for finding one or more assetsthat can be included in a story product. The assets, which meet therequirements and rules of the image product, are included.

U.S. Pat. No. 7,836,093, incorporated by reference herein in itsentirety, describes systems and methods for generating user profilesbased at least upon an analysis of image content from digital imagerecords. The image content analysis is performed to identify trends thatare used to identify user subject interests. The user subject interestscan be incorporated into a user profile that is stored in aprocessor-accessible memory system.

U.S. Patent Application Publication No. 2009/0297045, incorporated byreference herein in its entirety, teaches a method of evaluating a usersubject interest based at least upon an analysis of a user's collectionof digital image records and is implemented at least in part by adata-processing system. The method receives a defined user subjectinterest, receives a set of content requirements associated with thedefined user-subject-interest, and identifies a set of digital imagerecords from the collection of digital image records each having imagecharacteristics in accord with the content requirements. Asubject-interest trait associated with the defined user-subject-interestis evaluated based at least upon an analysis of the set of digital imagerecords or characteristics thereof. The subject-interest trait isassociated with the defined user-subject-interest in aprocessor-accessible memory.

U.S. Patent Application Publication No. 2007/0177805, incorporated byreference herein in its entirety, describes a method of searchingthrough a collection of images, includes providing a list of individualsof interest and features associated with such individuals; detectingpeople in the image collection; determining the likelihood for eachlisted individual of appearing in each image collection in response tothe people detected and the features associated with the listedindividuals; and selecting in response to the determined likelihoods anumber of images such that each individual from the list appears in theselected images. This enables a user to locate images of particularpeople.

U.S. Pat. No. 6,389,181, incorporated by reference herein in itsentirety, discusses photo-collage generation and modification usingimage processing by obtaining a digital record for each of a pluralityof images, assigning each of the digital records a unique identifier andstoring the digital records in a database. The digital records areautomatically sorted using at least one date type to categorize each ofthe digital records according at least one predetermined criteria. Thesorted digital records are used to compose a photo-collage. The methodand system employ data types selected from digital image pixel data;metadata; product order information; processing goal information; or acustomer profile to automatically sort data, typically by culling orgrouping, to categorize images according to either an event, a person,or chronology.

U.S. Pat. No. 6,671,405, incorporated by reference herein in itsentirety, to Savakis, et al., entitled “Method for automatic assessmentof emphasis and appeal in consumer images,” discloses an approach whichcomputes a metric of “emphasis and appeal” of an image, without userintervention and is included herein in its entirety by reference. Afirst metric is based upon a number of factors, which can include: imagesemantic content (e.g. people, faces); objective features (e.g.,colorfulness and sharpness); and main subject features (e.g., size ofthe main subject). A second metric compares the factors relative toother images in a collection. The factors are integrated using a trainedreasoning engine. The method described in U.S. Patent ApplicationPublication No. 2004/0075743 by Chantani et al., entitled “System andmethod for digital image selection”, incorporated by reference herein inits entirety, is somewhat similar and discloses the sorting of imagesbased upon user-selected parameters of semantic content or objectivefeatures in the images. U.S. Pat. No. 6,816,847 to Toyama, entitled“Computerized aesthetic judgment of images”, incorporated by referenceherein in its entirety, discloses an approach to compute the aestheticquality of images through the use of a trained and automated classifierbased on features of the image. Recommendations to improve the aestheticscore based on the same features selected by the classifier can begenerated with this method. U.S. Patent Application Publication No.2011/0075917, incorporated by reference herein in its entirety,describes estimating aesthetic quality of digital images and isincorporated herein in its entirety by reference. These approaches havethe advantage of working from the images themselves, but arecomputationally intensive.

While these methods are useful for sorting images into event groups,temporally organizing the images, assessing emphasis, appeal, or imagequality, or recognizing individuals in an image, they do not address theneed for automating the selection of a narrative structure or digitalimages from a large collection of images to provide a selection of avariety of images that provide interest, aesthetic appeal, and anarrative structure. Selecting images from a large image collection tomatch a narrative structure can be difficult, even with the use ofimaging tools or personalization information. It can also be difficultto select a narrative structure. For example, a user might desire tomake an image-based product using images from an available digital imagecollection, but be unaware of what narratives might be supported by theavailable images in the collection.

There is a need therefore, for an improved automated method forselecting a narrative structure and digital images from a largecollection of digital images to provide a selection of a variety ofimages that provide interest, aesthetic appeal, and narrative structurein an image-based product.

SUMMARY OF THE INVENTION

In accordance with the present invention, a method is provided formaking an image-based product comprising:

providing an image collection having a plurality of digital images, eachdigital image having an image type;

providing one or more image-type distributions, each image-typedistribution corresponding to a theme and including a distribution ofimage types related to the theme;

using a processor to automatically compare the image types of thedigital images in the image collection to the image types in theimage-type distribution;

using the processor to automatically determine a match between the imagetypes in the image collection and the image types in the image-typedistribution; selecting a group of digital images from the imagecollection having a distribution of image types specified by thedetermined matching image-type distribution;

assembling the digital images in the selected group of images into animage-based product; and

causing the construction of the image-based product.

The present invention provides an improved automated method forselecting a narrative structure and digital images from a largecollection of digital images to provide a selection of a variety ofdigital images that provide interest, aesthetic appeal, and narrativestructure in an image-based product.

These, and other, aspects of the present invention will be betterappreciated and understood when considered in conjunction with thefollowing description and the accompanying drawings. It should beunderstood, however, that the following description, while indicatingpreferred embodiments of the present invention and numerous specificdetails thereof, is given by way of illustration and not of limitation.For example, the summary descriptions above are not meant to describeindividual separate embodiments whose elements are not interchangeable.In fact, many of the elements described as related to a particularembodiment can be used together with and interchanged with, elements ofother described embodiments. Many changes and modifications can be madewithin the scope of the present invention without departing from thespirit thereof, and the invention includes all such modifications. Thefigures below are not intended to be drawn to any precise scale withrespect to relative size, angular relationship, or relative position orto any combinational relationship with respect to interchangeability,substitution, or representation of an actual implementation.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features, and advantages of the present inventionwill become more apparent when taken in conjunction with the followingdescription and drawings wherein identical reference numerals have beenused to identify identical features that are common to the figures, andwherein:

FIGS. 1-5 are flow diagrams according to various embodiments of thepresent invention;

FIG. 6 illustrates a histogram of an image-type distribution useful inunderstanding the present invention;

FIG. 7 illustrates a histogram of an image-type distribution useful inunderstanding the present invention;

FIG. 8 illustrates a flow diagram according to another embodiment of thepresent invention;

FIG. 9 illustrates a 100% stacked column chart of an image typedistribution useful in understanding the present invention;

FIG. 10 illustrates another 100% stacked column chart of an image typedistribution useful in understanding the present invention;

FIG. 11 illustrates a relative frequency column chart of an image-typedistribution useful in understanding the present invention;

FIG. 12 illustrates a flow diagram according to another embodiment ofthe present invention;

FIGS. 13A and 13B illustrate 100% stacked column charts of two differentdistributions of identified persons useful in understanding the presentinvention;

FIG. 14 is a simplified schematic of a computer system useful for thepresent invention;

FIG. 15 is a schematic of a computer system useful for embodiments ofthe present invention; and

FIG. 16 is a schematic of another computer system useful for embodimentsof the present invention.

DETAILED DESCRIPTION OF THE INVENTION

According to various embodiments of the present invention, a theme foran image-based product is selected, received, provided, or automaticallydetermined based on digital images in an image collection. The digitalimages have one or more image types. The theme provides a narrativestructure and is specified by a distribution of image types. Once thetheme and image-type distribution are determined, a group of digitalimages having image types corresponding to the image-type distributionfrom the image collection is automatically selected. The group ofdigital images is assembled into an image-based product and theimage-based product constructed.

Referring to FIG. 1, an image collection having a plurality of digitalimages is provided in step 500. Each digital image has an image typeassociated therewith. In step 505, one or more image-type distributionsis provided. Each image-type distribution corresponds to one or morethemes and includes a distribution of image types related to thetheme(s). A processor or computer automatically compares the image typesof the digital images in the image collection to the image typesspecified in the image-type distribution in step 510 and thenautomatically determines a match between the image types in the imagecollection and the image types in the image-type distribution (step511). If a match is not determined, the process can end since the imagecollection does not have suitable matching images with image typescorresponding to the image-type distributions. If a match is determined,a group of digital images having the distribution of image typesspecified by the determined matching image-type distribution is selectedfrom the image collection in step 515. An image-based product isselected (step 520) and the digital images in the selected group ofimages are assembled into the selected image-based product in step 525and the image-based product caused to be constructed in step 530. Instep 535, the image-based product can be delivered, for example to acustomer, user, or other recipient.

In embodiments of the present invention, the image-based productselection of step 520 is made on the basis of the image-typedistribution determined (as illustrated in FIG. 1). In alternativeembodiments, the image-based product selection is done by a userseparately from the image-type distribution selection. In suchalternative embodiments, the image-type distribution selection can belimited to those image-type distributions associated or compatible withthe selected image-based product.

In various embodiments, the image type associated with each image isincluded in an electronic database including the digital images orreferences to the digital images. Alternatively, each digital imageincludes metadata in an electronic digital image file, for example in afile header, specifying the image type. The image distributions can bestored as numbers in an electronic file, for example with an associationbetween each image type and a percentage value or range of values.Digital images, electronic image files, databases, and informationstored in files are all well known in the computing arts, together withcomputers, processors, and storage devices for accessing, storing andmanipulating such information. Software tools for writing programs thatmanipulate information are also well known.

According to various embodiments of the present invention, the processorcan automatically determine a match between the image types in the imagecollection and the image types in the image-type distribution. Referringto FIG. 2, the steps 510 and 511 of FIG. 1 are illustrated in a moredetailed example. After the image distribution is provided in step 505(FIG. 1), the image types for each image distribution are compared tothe digital images in the image collection in iterated step 900. Foreach digital image of iterated step 905, the image types of the iteratedimage distribution are compared to the image types of the digital image(step 910), for example by iteratively comparing each image type of theiterated image-type distribution to each image type of the iterateddigital image. Matching image types between the iterated digital imageand iterated image-type distribution are recorded in step 915. Once allof the image types for each of the digital images that match theiterated image distribution are recorded, the matches are combined (step920) to determine an overall match between the iterated image-typedistribution and the digital images in the image collection, for exampleby summing the number of matches of each image type found in theimage-distribution. If the number of matches for some image type in theimage-type distribution is zero, a match between the image-typedistribution and the digital images in the image collection isincomplete or not made.

An image-match metric is calculated to determine how well the digitalimages in the image collection match the image distribution in step 925.For example, the image-match metric could include the overall number ofmatching images, the average percentage match of the image types of eachdigital image with the image types of the image-type distribution, orthe overall variety of image types in the image collection that matchthe image-type distribution. A variety of image-match metrics can beemployed to provide a measure of the match between digital images in theimage collection and the image-type distribution.

The image-match metric is iteratively calculated for each of the imagedistributions provided. The image-match metrics are then compared instep 930 and the image-type distribution best matching the digitalimages in the image collection is selected in step 935. Once theimage-type distribution is selected, the digital images are selected instep 515 if a suitable match is found.

In various embodiments of the present invention, an image-typedistribution is not associated with a particular image-based product ortype of image-based product. Digital images selected from the imagecollection are applicable or can be used in a variety of differentimage-based products. In alternative embodiments of the presentinvention, an image-type distribution is associated with a particularimage-based product or type of image-based product. Digital imagesselected from the image collection are adapted for use in the associatedimage-based products.

As illustrated in FIGS. 1 and 2, if a suitable image-type distributionmatch is not found and no selection made, the process can stop. In otherembodiments, however, additional digital images can be provided to theimage collection. Referring to FIG. 3, one or more of the image-typedistribution matches are analyzed in step 600 to determine image typesmissing from the image collection present in the image-typedistribution. The missing image types are then communicated (step 605),for example to a user or customer. The user or customer receives themissing image types (step 606) and obtains digital images (step 607),for example from a different image collection. The obtained digitalimages having the missing image types are communicated (step 608), andreceived (step 610). The obtained digital images can be added to theimage collection and the image-type distribution and digital image typecomparisons repeated until a suitable match is determined. In this way,if the image types in an image collection are not present in one or moreimage-type distributions, the image collection can be reinforced withadditional digital images having image types corresponding to the imagetypes needed by the image-type distributions and to thereby enable theconstruction of a desired image-based product.

In embodiments of the present invention, only one image-typedistribution is provided. In those embodiments, a determination ofwhether a suitable match can be made is made. If no suitable match isavailable, additional digital images can be provided or a differentimage-type distribution (possibly corresponding to a differentimage-based product) provided. In alternative embodiments, multipleimage-type distributions are provided. In such alternatives, the bestmatch can be selected, used to specify which digital images areselected, and a corresponding image-based product constructed. Referringto FIG. 4, the best match determined can be approved by a user, ifdesired. As shown in FIG. 4, image-type distributions are provided instep 705 and compared with the image types of the digital images in theimage collection (step 710). The matching image-type distributions arecompared (step 711) and the best match selected (step 712, for exampleas described with respect to FIG. 2). Optionally, the best image-typedistribution match is communicated (step 713), for example to a user orcustomer who then reviews the selected best image-type distribution andresponds with a match approval that is then received in step 714. If theimage-type distribution match is not approved and received, the processcan stop with no further action (or further digital images can besolicited as discussed with respect to FIG. 3). Digital imagescorresponding to the received image-type distribution match are thenselected in step 515.

In other embodiments, referring to FIG. 5, a plurality of image-typedistribution matches is communicated and a preferred match selectionreceived. As shown in FIG. 5, image-type distributions are provided instep 705 and compared with the image types of the digital images in theimage collection (step 710). Optionally, the matching image-typedistributions are compared (step 811) and ordered (step 812), forexample by applying a metric such as that used for comparing. Thematches (whether ordered or not) are communicated (step 813), forexample to a user or customer, and a selected image-type distributionmatch is received in step 814. Digital images corresponding to thereceived image-type distribution match are then selected in step 515.

In other embodiments, a theme can be received and one or more image-typedistributions received, provided, or selected on the basis of the theme.The image types of the selected image-type distributions are thencompared to the image types of the digital images in the imagecollection. Image-type distributions can also be received or provided.In either case, a user or customer can select the theme or theimage-type distributions whose image types are compared with the imagetypes of the digital images in the image collection.

In yet other embodiments, the digital images in the collection can beanalyzed to determine one or more themes. Image types of image-typedistributions corresponding to the determined themes can then becompared with the image types of the digital images in the imagecollection. In another embodiment, one or more image-type distributionsassociated with an image-based product are provided. For example, a useror customer can select a desired image-based product. Image-typedistributions corresponding to the image-based product can then be usedto automatically select digital images, as described above.

In any or all of these embodiments, a user or customer can communicatean image collection or select the desired theme, image-typedistributions, or image-based products. The digital images, selectedtheme, image-type distributions, or image-based products can becommunicated, for example via the internet, from a computer (e.g. apersonal computer at a user's or customer's home) to a computer serverthat stores the image collection, image-type distributions, andimage-based product information. The computer server compares the imagetypes of the digital images in the image collection to the image typesin a specified image-type distribution by accessing the image types ofeach digital image and comparing the accessed image types to theimage-type distribution, determines a match, and selects digital images,as described above. Useful personal computers, server computers,communication network devices, and software tools for making programsthat implement the steps described are all known in the art and arediscussed in greater detail below.

According to the present invention, an image-based product is a printedor electronic product that includes multiple images incorporated into animage-related object (either real or virtual), such as for example aphoto-book, photo-album, a photo-card, a picture greeting card, aphoto-collage, a picture mug, or other image-bearing item. Theimage-based product can be printed on a substrate or stored in andretrieved from an electronic storage system. The images can be a user'spersonal images and the image product can be personalized. The imagescan be located in specified pre-determined locations within theimage-based product or adaptively located according to the sizes, aspectratios, orientations and other attributes of the images. Likewise, theimage sizes, orientations, or aspect ratios included in the image-basedproduct can be adjusted, either to accommodate pre-defined templateswith specific pre-determined openings or adaptively adjusted forinclusion in an image-based product.

In an embodiment, the image-based product is a specified quantity ofdigital images, a specified size of digital images, or a specifiedresolution of digital images.

As intended herein, an image-based product can include printed images,for example images printed on photographic paper, cardboard, writingpaper, textiles, ceramics, or rubber such as foam rubber, and polymers.These printed images can be assembled or bound into an image-basedproduct, for example a book. In an alternative embodiment, theimage-based product can be an electronic image-based product suitablefor display on an electronic display by a computing device and stored asa file, or multiple files, in an electronic storage system such as acomputer-controlled disk drive or solid-state memory. Such image-basedproducts can include, for example, photobooks, collages, videos, orslide shows that include one or more images with or without ancillaryimages such as templates, backgrounds, clip art and the like. In variousembodiments, an image-based product includes a single still image,multiple still images, or video images and can include other sensorymodalities such as sound. The electronic image-based products aredisplayed by a computer on a display, for example as a single image orby sequentially displaying multiple pages in the image-based producttogether with outputting any other related image product informationsuch as sound. Such display can be interactively controlled by a user.Such display devices and image-based products are known in the art asare user interfaces for controlling the viewing of image-based productson a display.

Similarly in various embodiments, image-type distributions and digitalimages in an image collection are stored as a file, or multiple files,in an electronic storage system such as a computer-controlled disk driveor solid-state memory. The storage medium can be connected directly to acomputer or available over a network, such as a local area network orthe internet. The stored information is accessed and manipulated by aprocessor that has access to the storage medium

A theme, as used in the present invention, is a narrative structurehaving a unifying subject. A narrative, for example, can be a story oraccount of events or experiences and associated, for example, with aperson, group, object, or location. A narrative structure has elementscorresponding to elements of the narrative. Elements are represented byimages in the image-based product. Elements can include actions,characters, objects, locations, or events. A theme is a story line thatcan be associated with an event.

For example, the theme of an event is a birthday party (a unifyingsubject) that has a narrative structure including guest arrival, giftpresentation, game playing, guest snack, lighting candles on a cake,blowing out candles on a cake, singing, gift opening, and guestdeparture. A theme can also be a macro-event that includes other events,for example, a history of primary school for a student, including eventsof each grade. In this example, the unifying subject is the studenthistory and the narrative structure can include, for each grade, aschool picture, a casual picture of the student with friends, and animage of a school event in which the student participates.

An image-based product is a product that includes multiple images. Theplurality of digital images can form an image collection and can bestored in an electronic storage and retrieval system, for example aprocessor-controlled rotating magnetic- or optical-media disk or asolid-state memory. The digital images have attributes that arespecified as image types. An image-type distribution is a specificationfor a set of digital images, each digital image having one or more imagetypes, whose digital images match a statistical distribution of imagetypes. An image type can be an attribute of an image. An imagecollection from which digital images are selected by a processor caninclude digital images from a variety of events, capture times, capturelocations, either related or unrelated, and can correspond to differentthemes, either related or unrelated. An image collection can includedigital images that correspond to multiple themes. A set of digitalimages or selected digital images within a set can correspond tomultiple related themes.

Selected digital images have image types that correspond to one or moreelements of a theme. The image-type distribution corresponds to aselected theme. For example, an image-type distribution can include atleast one of each element of a narrative structure and can includemultiple digital images of a specific element. For example, if the themeis a history of primary school for a student, selected digital imagescan have one or more image types such as school picture, group photo,activity, person identity, and capture time.

Referring to FIG. 6, a histogram of a digital image collection having aplurality of digital images of four different image types isillustrated. This kind of histogram profile is also referred to hereinas an image-type distribution. An image-type distribution can be used todescribe a collection of digital images in a database (collection) ofimages or in an image-based product, and it can be used as a filter ortemplate to predefine a distribution of digital images, which is thenused to select digital images from an image collection (or database) tobe included in an image-based product. The height of each columnindicates an image type count 300 of digital images of the image typemarked. In this example, the largest plurality of the digital images isof image type 4, followed by digital images of image type 2 and thendigital images of image type 1. The fewest digital images are of imagetype 3.

As another example, a digital image collection containing one hundreddifferent digital images classified into four image types of twenty-fivedigital images each has an image-type distribution that is equivalent toa collection of four images with one each of the four exclusive imagetypes, because both distributions contain 25% each of four image types.Thus, the one-hundred-image collection can produce twenty-five uniquegroups of images having the same image-type distribution as the originalcollection without any image repeated in any of the groups.

Referring again to the example of a history theme of a student'sexperiences in a primary school, FIG. 7 illustrates an example of animage-type distribution having image type counts 300 in a histogramwhose corresponding digital images can be selected to narrate the theme.Other image-type distributions can be employed. In this example, thestudent has nine years of experience in a primary school for gradeskindergarten through eight. 18 subject images, 9 single person images, 9group images, and 9 event images are required in the image-typedistribution, as well as three images per year whose capture datecorresponds to the years of historical interest. This image-typedistribution can be satisfied by selecting one image of the subject in asingle person image, one image of the subject in a group image, and oneimage of an event for each of the nine years (three images per year).Note that an image can have multiple types; as illustrated here asubject image has a date image type, a subject image type, and a singleor group image type. An image-type distribution can have optionalelements, for example the 18 images of the subject can include at least18 images or exactly 18 images. If the number is exactly 18, the eventimages do not include the subject. If the number is at least 18, theevent images can, but do not have to, include the subject.

Many other image types can be included, including different subjects.For example, events are categorized by event type (e.g. athletic,musical, theatre, field trip) and an image-type distribution can requireat least a specified percentage or no more than a specified percentageof images having image types corresponding to the event types.

The steps illustrated in FIGS. 1-5 are performed, for example, by aprogrammable processor executing a software program and connected to amemory storage device, for example an electronic storage system, asdescribed further below. The processor can be a standalone computer,e.g. a desktop computer, a portable computer, or a server computer.Alternatively the processor can be a networked computer capable ofcommunicating with other networked computers and the tasks of thepresent invention are cooperatively performed by multiple interactingprocessors. The network is, for example, the internet or a cellulartelephone network. In one embodiment, the steps of the present inventionare performed with a client—server computer network. Such processors,computer systems, and communication networks are known in the computingindustry.

A user can communicate from a remote location to provide digital imagesin the image collection, themes, or image-type distributions. Referringto FIG. 8, in further embodiments of the present invention, theplurality of digital images is received in step 550, for example,through a network, from an image source, for example a digital camera,remote client computer, or portable computer connected to a cellulartelephony or WiFi network. In various embodiments, the plurality ofdigital images is stored in a permanent non-volatile storage device,such as rotating magnetic media or the plurality of digital images isstored in a volatile memory, for example, random access memory (RAM).Likewise, in an embodiment, the theme and image-type distributions arereceived from a remote source. In an embodiment, such a remote source isowned by a user and the processor is owned by a service provider.Alternatively, the theme and image-type distributions are stored in amemory under the control of the processor.

In one embodiment of the present invention, the image types are providedwith the digital images. For example, a user might assign image types todigital images. In another embodiment, the image types are associatedwith the digital images by processing and analyzing (step 555) thedigital images to determine the image type(s) of the digital images(step 560), for example, by analyzing the digital images using softwareexecuting mathematical algorithms on an electronic processor. Suchmathematics, algorithms, software, and processors are known in the art.Alternatively, the image types are determined manually, for example, byan owner of the digital images interacting with the digital imagesthrough a graphic interface on a digital computer and providing metadatato the processing system which is stored therein. The metadata can bestored in a metadata database associated with the digital imagecollection or with the digital image itself, for example in a fileheader.

A theme can be selected by receiving a theme selection from a user orthe theme selected by an analysis of the plurality of digital images.The present invention is explicitly intended to include both or eithermethod of theme selection. In various embodiments, a theme is selectedby receiving it from a user and an image-type distribution chosen by theprocessor.

Thus, as further illustrated in FIG. 8, themes can be selected, e.g.received from a user or owner of the digital images (step 565) orprovided through an analysis of the digital images to determine a theme.In such an embodiment, the method further includes analyzing theplurality of digital images (step 570), to determine themes associatedwith the plurality of digital images (step 575). Similarly, image-typedistributions can be provided, e.g. received from a user or owner of thedigital images (step 580). In various embodiments, image-typedistributions are dependent on the plurality of digital images, themes,or a desired image-based product. The themes or image-type distributionsdetermined are generated and stored, for example in an electronicstorage system controlled by the processor. The determined themes orimage-type distributions can also be communicated to a user or owner ofthe digital images.

For example, an image collection including an image of a cake with litcandles and “Happy Birthday” written on the cake is deduced to includeimages of a birthday party and a birthday party theme selected. If animage of a person blowing out the candles is found in the collection,the person is deduced to be the main subject of the party. U.S. PatentApplication Publication 2008/0304808 discloses semantic methods fordetermining themes and automatically classifying events.

In a further embodiment, the processor can cause the construction of theimage-based processor, for example by printing the selected digitalimages, by sending the selected digital images to a third party forprinting, or by making an electronic image-based product such as a slideshow, video, photo-book, or collage and storing the electronicimage-based product in an electronic storage system. An electronicimage-based product is sent to a user electronically or a printedimage-based product is sent to a user by physical delivery, e.g. througha postal or package delivery service. Alternatively, the image-basedproduct is delivered to a third party, for example as a gift.

The image collection of digital images can have digital imagescorresponding to different themes. A large image collection can includeimages of many different, unrelated events that can overlap in time.Different subsets of the digital images in the image collection cancorrespond to different themes. In other embodiments, a single subset ofdigital images can have different themes that, for example, cancorrespond to different perspectives on the information recorded in thesubset of digital images. A single theme can have different image-typedistributions corresponding to different ways of communicating thenarrative structure inherent in the theme. In other embodiments,different image-based products can correspond to different image-typedistributions, for example in number of images or image types.Image-based products can be limited to particular image-typedistributions or vice versa.

If an image collection does not have the digital images corresponding toan image-type distribution, an alternative distribution corresponding tothe selected theme can be selected, additional digital images procured,or a different image-based product chosen.

Image types can explicitly correspond to narrative structural elementsof themes. For example, image types can include introduction type,character type, person type, object type, action type, and conclusiontype. The digital images can have a temporal association and theimage-type distribution include an image type time order correspondingto a temporal order in the narrative structure. To support this, animage type can be a capture time of the corresponding digital image.

Since many themes are organized around specific individuals or groups ofindividuals, image-type distributions can include a specifieddistribution of image types of specific persons or character types. Animage type can include an identified person type and the digital imagescan be analyzed to recognize and identify persons in an image. Theidentified person can correspond to an image type.

An image type is a category or classification of image attributes andcan be associated with a digital image as image metadata stored with thedigital image in a common electronic file or associated with the digitalimage in a separate electronic file. An image can have more than oneimage type. For example, a digital image can have an image type such asa portrait orientation type, a landscape orientation type, or a scenicimage type. The same digital image can also be classified as an imagethat includes a person type, a close-up image of a person type, a groupimage that includes multiple people type, day-time image type,night-time image type, image including one or more animals type,black-and-white image type, color image type, identified person type,identified gender type, and flash-exposed image type. An image type canbe an image-usage type classifying the digital image as a popular imageand frequently used. Other types can be defined and used as needed forparticular image products or as required for desired image-typedistributions. Therefore, a variety of digital images having a desireddistribution of image types such as those listed above can be selected.

An image type can include a value that indicates the strength or amountof a particular type for a specific image. For example, an image is agroup image but, if it only includes two people, the strength of thegroup-type is relatively weak compared to a group image that includes 10people. In this example, an integer value representing a number ofpersons appearing in the digital image is stored with or in associationwith the digital image to indicate its group-type strength or value. Asan example of ranking group-type digital images, a collection of theseimages is sorted in descending order according to a magnitude of theirgroup-type value. A selection algorithm for finding images depicting agroup can be programmed to preferably select images with a highergroup-type value by preferably selecting images from the top of thesorted list.

An image-usage type can have a strength value indicating how often orhow much the corresponding digital image is used, for example includinga combination of metrics such as how often the image is shared orviewed, whether the image was purchased, edited, used in products, orwhether it was deleted from a collection. Alternatively, each of thoseattributes can be a separate image-type classification. The image-usagetype(s) can indicate how much a user values the corresponding digitalimage. As an example ranking method, the number of times that an imagefile was opened, or an image shared or viewed is accumulated for eachimage and then the images ranked in descending order according to thenumber. A preferential selection scheme can then be implemented wherebythe images listed at the top of the ranking are preferentially selected.

An image type can also include a similarity metric that indicates therelative uniqueness of the image. For example, if an image is verydifferent from other images, it can have a high uniqueness image-typevalue (or an equivalent low similarity value). If an image is similar toone or more of the other images, it can have a low uniqueness image-typevalue (or an equivalent high similarity value) depending on the degreeof similarity and the number of images to which it is similar. Thus,every image can have the same image type but with varying values. Theimage-type value can also be associated with a digital image as imagemetadata stored with the digital image in a common electronic file orassociated with the digital image in a separate electronic file.

For example, a first desired image-type distribution specification caninclude 20% scenic images, 60% scenic images that include a person, and20% close-up images. The actual number of images of each type is thencalculated by multiplying the total number of images in the desiredimage-based product by the percentage associated with the image type inthe desired image-type distribution. The total number of digital imagesin the image-based product is determined by the image-based product tobe used. A desired image-type distribution can also include multiplevalues corresponding to an image type that has multiple values ratherthan a simple binary classification value.

Referring to FIGS. 9 and 10, two different desired distributions ofimage types are illustrated in a 100% stacked column chart in which thetotal number of image types is 100%. In FIG. 6, a percent image-typedesired image-type distribution 320 of image type 4 is largest, similarto the desired distribution of image types in the collection. However,the prevalence of image type 3 in the desired image-type distribution isrelatively smaller than in the collection and the prevalence of imagetypes 1 and 2 in the desired image-type distribution are equal. Thus,according to the example of FIG. 6, the desired distribution of imagetypes in an image-based product has relatively fewer digital images ofimage type 2 and 3 than are in the original collection.

Referring to the example of FIG. 10, the percent image-type desireddistribution 320 of image types 2 and 4 are relatively reduced while thepercent image-type desired distribution of image types 3 and 1 areincreased.

Because a digital image can have multiple image types, a desireddistribution need not have a relative frequency of digital images thatadds to 100%. For example an image is a landscape image, a scenic image,and a scenic image that includes a person. Similarly, a close-up imageis a portrait image and a flash image. Thus, in a second example, asecond desired distribution can include 10% scenic images, 40% landscapeorientation, 80% day-time image, 100% color image, 60% scenic image thatincludes a person, and 20% close-up image. In an alternative embodiment,the image types are selectively programmed to be mutually exclusive sothat no image is determined to have more than one image type. In thisinstance the relative distribution percentages should add up to 100%.

Referring to FIG. 11, a desired distribution of image types 320 isillustrated in which the relative frequency of each distribution ofimage type 320 is shown by the height of the corresponding column. Therelative frequency ranges from 0% (not desired in any selected digitalimage) to 100% (desired in all selected digital images).

In another embodiment of the present invention, a desired image-typedistribution can include more than, but not fewer than, the specifiedrelative frequency of image types. This simplifies the task of selectingdigital images when a digital image has more than one image type. Forexample, if a desired image-type distribution requires a certainrelative frequency of close-up images and a different relative frequencyof portrait images, a close-up image that is also a portrait image isselected, even if the relative frequency of portrait images in a desiredimage-type distribution is then exceeded. In various preferredembodiments of the present invention, variation in the relativefrequency of images of specified image types is controlled, for examplewithin a range such as from 60% to 80% range or 60% to 100%. Rules canbe associated with the digital image selection to control the imageselection process in accordance with the desired image-typedistribution, for example specifying a desired degree of flexibility inselecting images that have multiple image types.

According to further embodiments of the present invention, digitalimages are automatically selected from the plurality of digital imagesto match the desired image-type distribution, for example by using aprocessor for executing software and an electronic storage storingdigital images, theme selections, and image-type distributions.

According to yet another embodiment of the present invention, differentdesired distributions of digital images in a common plurality of digitalimages are specified for multiple image-based products. The same themeis then used for different image-based products for differentindividuals but with image-type distributions specifying differentindividuals. For example, if multiple people take a scenic vacationtogether, a commemorative photo-album for each person can be createdthat emphasizes images of different image types preferred by that personspecified by different digital image desired distributions and thatincludes the corresponding subject. Thus, the same collection of digitalimages can be used to produce multiple image-based products havingdifferent image-type desired distributions, for example for differentintended recipients of the photo-products.

In another example, a person might enjoy a beach vacation and wish tospecify a photo-product such as a photo-album for each of his or herparents, siblings, friends, and others. In each photo-album, arelatively greater number of pictures including the recipient can beprovided. Thus, a different selection of digital images is specified bya different desired distribution of digital images.

In one embodiment of the present invention, the various methods of thepresent invention are performed automatically using, for example,computer systems such as those described further below. Ways forreceiving images, photo-product choices, and desired distributions, e.g.using communication circuits and networks, are known, as are ways formanually selecting digital images and specifying photo-products, e.g. byusing software executing on a processor or interacting with an on-linecomputer server.

A method of the present invention can further include the steps ofremoving bad images from an image collection, for example by analyzingthe images to discover duplicate images or dud images. A duplicate imagecan be an exact copy of an image in the plurality of images, a copy ofthe image at a different resolution, or a very similar image. A dudimage can be a very poor image, for example an image in which the flashfailed to fire or was ineffective, an image in which the camera lens ofan image-capturing camera was obscured by a finger or other object, anout-of-focus image, or an image taken in error.

In a further embodiment of the present invention, the image quality ofthe digital images in the plurality of digital images is determined, forexample by analyzing the composition, color, and exposure of the digitalimages, and ranked. A similarity metric can also be employed describingthe similarity of each digital image in the plurality of digital imagesto every other digital image in the plurality of digital images. Qualityand similarity measures are known in the art together with softwareexecuting on a processor to determine such measures on a collection ofdigital images and can be employed to assist in the optional duplicationand dud detection steps and to aid in the image-selection process. Forexample, if a desired distribution requires a close-up, portrait imageof a person and several such digital images are present in the pluralityof digital images, the digital image having the best image quality andthe least similarity to other digital images can be chosen. The selectedimages then specify the photo-product. The similarity and quality valuescan be associated with a digital image as image metadata stored with thedigital image in a common electronic file or associated with the digitalimage in a separate electronic file. Once the number and types ofdigital images are selected, the specified image-based product can belaid out and completed, as is known by practitioners in the art, andthen caused to be manufactured and delivered to a recipient.

Image types can include images having persons therein or images havingspecific individuals therein. Face recognition and identification can beperformed manually on an image, for example by a user, and theinformation stored as a corresponding image type. Face recognition andidentification can also be done automatically. Using computer methodsdescribed in the article “Rapid object detection using a boosted cascadeof simple features,” by P. Viola and M. Jones, in Computer Vision andPattern Recognition, 2001, Proceedings of the 2001 IEEE Computer SocietyConference, 2001, pp. I-511-I-518 vol. 1; or in “Feature-centricevaluation for efficient cascaded object detection,” by H. Schneiderman,in Computer Vision and Pattern Recognition, 2004; Proceedings of the2004 IEEE Computer Society Conference, 2004, pp. II-29-II-36, Vol. 2.,the size and location of each face can be found within each digitalimage and is useful in determining close-up types of images and imagescontaining people. These two documents are incorporated by referenceherein in their entirety. Viola uses a training set of positive face andnegative non-face images. The face classification can work using aspecified window size. This window is slid across and down all pixels inthe image in order to detect faces. The window is enlarged so as todetect larger faces in the image. The process repeats until all faces ofall sizes are found in the image. Not only will this process find allfaces in the image, it will return the location and size of each face.

Active shape models as described in “Active shape models—their trainingand application,” by Cootes, T. F. Cootes, C. J. Taylor, D. H. Cooper,and J. Graham, Computer Vision and Image Understanding, vol. 61, pp.38-59, 1995, can be used to localize all facial features such as eyes,nose, lips, face outline, and eyebrows. These documents are incorporatedby reference herein in their entirety. Using the features that are thusfound, one can then determine if eyes/mouth are open, or if theexpression is happy, sad, scared, serious, neutral, or if the person hasa pleasing smile. Determining pose uses similar extracted features, asdescribed in “Facial Pose Estimation Using a Symmetrical Feature Model”,by R. W. Ptucha, A. Savakis, Proceedings of ICME—Workshop on MediaInformation Analysis for Personal and Social Applications, 2009, whichdevelops a geometric model that adheres to anthropometric constraints.This document is incorporated by reference herein in its entirety. Withpose and expression information stored for each face, preferredembodiments of the present invention can be programmed to classifydigital images according to these various detected types (happy, sad,scared, serious, and neutral).

A main subject detection algorithm, such as the one described in U.S.Pat. No. 6,282,317, which is incorporated herein by reference in itsentirety, involves segmenting a digital image into a few regions ofhomogeneous properties such as color and texture. Region segments can begrouped into larger regions based on such similarity measures. Regionsare algorithmically evaluated for their saliency using two independentyet complementary types of saliency features—structural saliencyfeatures and semantic saliency features. The structural saliencyfeatures are determined by measureable characteristics such as location,size, shape and symmetry of each region in an image. The semanticsaliency features are based upon previous knowledge of knownobjects/regions in an image which are likely to be part of foreground(for example, statues, buildings, people) or background (for example,sky, and grass), using color, brightness, and texture measurements. Forexample, identifying key features such as flesh, face, sky, grass, andother green vegetation by algorithmic processing are well characterizedin the literature.

In one embodiment, once the image types are determined for each of thedigital images in the plurality of digital images, the relativefrequency of digital images of each image type can optionally bedetermined. For example, if a collection of 60 digital images isprovided and 30 are determined by the processing system to be scenic,then the relative frequency data stored in association with thecollection is a value representing 50%. This information is useful whenselecting the digital images from the collection to satisfy a specifiedimage-based product.

The relative frequency of image types in an image collection can also beoptionally used by selecting the image-based product to have a desireddistribution dependent on the relative frequency of image types in animage collection, since a given image-based product (e.g. auser-selected photo-product) can require a certain number of image typesof digital images in a collection. The desired distribution can have anequivalent image-type distribution to the image-type distribution of theimage collection, for example without repeating any digital images.Therefore, an image-based product can be selected, suggested to a user,or modified depending on the relative frequency or number of digitalimages of each image type in a digital image collection.

Similarly, the relative frequency of image types can also be optionallyused to select the image-type distribution, since an image-typedistribution can require a certain relative frequency or number of imagetypes of digital images in a collection. If, for example, an image-basedproduct requires a certain number of images and a first image-typedistribution cannot be satisfied with a given image collection, analternative second image-type distribution is selected. A variety ofways to specify an alternative second image-type distribution can beemployed. For example, a second image-type distribution, including thesame image types but requiring fewer of each image type, is selected.Alternatively, a second image-type distribution including image typesrelated to the image types required by the first distribution (e.g. agroup image with a different number of people) is selected. Therefore, adistribution can be selected depending on the relative frequency ornumber of digital images of each image type in a collection.

An image-based product having an image-type distribution (and a themeand intended audience) can thus be suggested to a user, depending on therelative frequency or number of image types in a digital imagecollection. Therefore, according to a method of the present invention, adifferent desired distribution is specified, received, or provided foreach of a variety of different audiences or recipients. An imagecollection can be analyzed and the analysis used to select a themesuggested to a user.

An image type of digital image can be an image with an identifiedperson. For example, an image type is a digital image including aspecific person, for example the digital image photographer, acolleague, a friend, or a relative of the digital image photographer asidentified by image metadata. Thus a distribution of digital images in acollection can include a distribution of specified individuals and avariety of the digital images that include a desired distribution ofpersons can be selected. For example, a variety of the digital imagescan include a desired distribution of close-up, individual, or groupimages including a desired person.

Thus, an embodiment of the present invention includes analyzing thedigital images to determine the identity of persons found in the digitalimages, forming one or more desired distributions of digital imagesdepending on each of the person identities, selecting a variety of thedigital images each satisfying the desired distribution, and specifyinga photo-product that includes each of the selected varieties of digitalimages.

Referring to FIG. 12, automatically determining image types in step 560can include analyzing a digital image (step 250) to determine theidentity of any persons in the digital image (step 255). Algorithms andsoftware executing on processors for locating and identifyingindividuals in a digital image are known. Thus, an image-based productcan be selected that includes a desired distribution of images ofspecific people. For example, at a family reunion, it might be desiredto specify a distribution of image types that includes a digital imageof at least one of every member of the family. If 100 digital images aretaken, then the distribution can include 1% of the image types for eachmember. If 20 family members are at the reunion, this distribution thenrequires that 20% of the pictures are allocated to digital images ofmembers (excluding group images). Depending on rules that are associatedwith the image selection process, a balance can be maintained betweennumbers of digital images of each family member in the specifiedphoto-product. Likewise, the number of individual or group images can becontrolled to provide a desired outcome. If the desired distributioncannot be achieved with the provided plurality of digital images, thedetermination of the relative frequency of image types can demonstratethe problem and an alternative image-based product or distributionselected or suggested. Since automated face finding and recognitionsoftware is available in the art, in an embodiment of the presentinvention, one can simply require that an image-based product include atleast one image of each individual in a digital image collection, thusindirectly specifying a distribution. Such an indirect distributionspecification is included as a specified distribution in an embodimentof the present invention.

Referring to FIGS. 13A and 13B, desired relative frequencies ofindividual image types for two different distributions 320 areillustrated. In FIG. 13A, persons A and B are desired to be equallyrepresented in the distribution of selected digital images, while personC is desired to be represented less often. In FIG. 13B, person B isdesired to be represented in the selected digital images more frequentlythan person A, and person C is not represented at all.

Since images frequently include more than one individual, it can bedesirable, as discussed above to include a distribution selection rulethat includes a desired number of images having the individual, or thatcontrols the number of group images versus individual images. Thus, aperson can be included in a desired number of selected images, selectedindividual images, or selected group images.

Users can specify image-type distributions using a computer, for examplea desktop computer known in the prior art. A processor can be used toprovide a user interface, the user interface including controls forsetting the relative frequencies of digital images of each image type.Likewise, a preferred method of the present invention can include usinga processor to receive a distribution of image types that includes arange of relative frequencies of image types.

In any of these embodiments, the digital image can be a still image, agraphical element, or a video image sequence, and can include an audioelement. The digital images can be multi-media elements.

Users can interact with a remote server with a client computer through acomputer network, such as the internet. The user can send the pluralityof image to the remote server, where it is stored. The user can alsoprovide an image-based product selection, a theme selection, andimage-type distribution selection, as desired. Images based on theselected theme and image-type distribution are selected and animage-based product is assembled from the selected images. Theimage-based product can be printed and delivered or made into anelectronic product and delivered electronically, for example by email,and viewed on a display screen by a user or other recipient.

In one embodiment of the present invention, a computer system for makingan image-based product includes a computer server connected to acommunication network for receiving communication from a remote clientcomputer; and a computer program. The computer program causes thecomputer server to store a plurality of digital images; provide one ormore image-type distributions, each image-type distributioncorresponding to a theme and including a distribution of image typesrelated to the theme; select a theme having a corresponding image-typedistribution, the image-type distribution having a distribution of imagetypes; use a computer to select digital images from the plurality ofdigital images, the selected digital image having the image-typedistribution corresponding to the selected theme; and assemble theimages in the selected group of images into an image-based product.

Various embodiments of the present invention can be implemented using avariety of computers and computer systems illustrated in FIGS. 14, 15and 16 and discussed further below. In one preferred embodiment, forexample, a desktop or laptop computer executing a software applicationcan provide a multi-media display apparatus suitable for providingimage-type distributions, providing digital image collections, orphoto-product choices, or for receiving such. In a preferred embodiment,a multi-media display apparatus includes: a display having a graphicuser interface (GUI) including a user-interactive GUI pointing device; aplurality of multi-media elements displayed on the GUI, and userinterface devices for providing a way for a user to enter informationinto the system. A desktop computer, for example, can provide such anapparatus.

In another preferred embodiment, a computer server can provide web pagesthat are served over a network to a remote client computer. The webpages can permit a user of the remote client computer to provide digitalimages, photo-product, or theme choices. Applications provided by theweb server to a remote client can enable presentation of selectedmulti-media elements, either as stand-alone software tools or providedthrough html, Java, or other known internet interactive tools. In thispreferred embodiment, a multi-media display system includes: a servercomputer providing graphical user interface display elements andfunctions to a remote client computer connected to the server computerthrough a computer network such as the internet, the remote clientcomputer including a display having a graphic user interface (GUI)including a user-interactive GUI pointing device; and a plurality ofmulti-media elements stored on the server computer, communicated to theremote client computer, and displayed on the GUI.

Computers and computer systems are stored program machines that executesoftware programs to implement desired functions. According to apreferred embodiment of the present invention, a software programexecuting on a computer with a display and graphic user interface (GUI)including a user-interactive GUI pointing device includes software fordisplaying a plurality of multi-media elements having images on the GUIand for performing the steps of the various methods described above.

FIG. 14 is a high-level diagram showing the components of a systemuseful for various embodiments of the present invention. The systemincludes a data processing system 110, a peripheral system 120, a userinterface system 130, and a data storage system 140. The peripheralsystem 120, the user interface system 130 and the data storage system140 are communicatively connected to the data processing system 110. Thesystem can be interconnected to other data processing or storage systemthrough a network, for example the internet.

The data processing system 110 includes one or more data processingdevices that implement the processes of the various preferredembodiments of the present invention, including the example processesdescribed herein. The phrases “data processing device” or “dataprocessor” are intended to include any data processing device, such as acentral processing unit (“CPU”), a desktop computer, a laptop computer,a mainframe computer, a personal digital assistant, a Blackberry™, adigital camera, a digital picture frame, cellular phone, a smart phoneor any other device for processing data, managing data, communicatingdata, or handling data, whether implemented with electrical, magnetic,optical, biological components, or otherwise.

The data storage system 140 includes one or more processor-accessiblememories configured to store information, including the informationneeded to execute the processes of the various preferred embodiments ofthe present invention, including the example processes described herein.The data storage system 140 can be a distributed processor-accessiblememory system including multiple processor-accessible memoriescommunicatively connected to the data processing system 110 via aplurality of computers or devices. On the other hand, the data storagesystem 140 need not be a distributed processor-accessible memory systemand, consequently, can include one or more processor-accessible memorieslocated within a single data processor or device.

The phrase “processor-accessible memory” is intended to include anyprocessor-accessible data storage device, whether volatile ornonvolatile, electronic, magnetic, optical, or otherwise, including butnot limited to, registers, caches, floppy disks, hard disks, CompactDiscs, DVDs, flash memories, ROMs, and RAMs.

The phrase “communicatively connected” is intended to include any typeof connection, whether wired or wireless, between devices, dataprocessors, or programs in which data is communicated. The phrase“communicatively connected” is intended to include a connection betweendevices or programs within a single data processor, a connection betweendevices or programs located in different data processors, and aconnection between devices not located in data processors at all. Inthis regard, although the data storage system 140 is shown separatelyfrom the data processing system 110, one skilled in the art willappreciate that the data storage system 140 can be stored completely orpartially within the data processing system 110. Further in this regard,although the peripheral system 120 and the user interface system 130 areshown separately from the data processing system 110, one skilled in theart will appreciate that one or both of such systems can be storedcompletely or partially within the data processing system 110.

The peripheral system 120 can include one or more devices configured toprovide digital content records to the data processing system 110. Forexample, the peripheral system 120 can include digital still cameras,digital video cameras, cellular phones, smart phones, or other dataprocessors. The data processing system 110, upon receipt of digitalcontent records from a device in the peripheral system 120, can storesuch digital content records in the data storage system 140.

The user interface system 130 can include a mouse, a keyboard, anothercomputer, or any device or combination of devices from which data isinput to the data processing system 110. In this regard, although theperipheral system 120 is shown separately from the user interface system130, the peripheral system 120 can be included as part of the userinterface system 130.

The user interface system 130 also can include a display device, aprocessor-accessible memory, or any device or combination of devices towhich data is output by the data processing system 110. In this regard,if the user interface system 130 includes a processor-accessible memory,such memory can be part of the data storage system 140 even though theuser interface system 130 and the data storage system 140 are shownseparately in FIG. 14.

Referring to FIGS. 15 and 16, computers, computer servers, and acommunication system are illustrated together with various elements andcomponents that are useful in accordance with various preferredembodiments of the present invention. FIG. 15 illustrates a preferredembodiment of an electronic system 20 that can be used in generating animage product or image-product specification. In the preferredembodiment of FIG. 15, electronic system 20 includes a housing 22 and asource of content data files 24, a user input system 26 and an outputsystem 28 connected to a processor 34. The source of content data files24, user-input system 26 or output system 28 and processor 34 can belocated within housing 22 as illustrated. In other embodiments, circuitsand systems of the source of content data files 24, user input system 26or output system 28 can be located in whole or in part outside ofhousing 22.

The source of content data files 24 can include any form of electronicor other circuit or system that can supply digital data to processor 34from which processor 34 can derive images for use in forming animage-enhanced item. In this regard, the content data files can include,for example and without limitation, still images, image sequences, videographics, and computer-generated images. Source of content data files 24can optionally capture images to create content data for use in contentdata files by use of capture devices located at, or connected to,electronic system 20 or can obtain content data files that have beenprepared by or using other devices. In the preferred embodiment of FIG.15, source of content data files 24 includes sensors 38, a memory 40 anda communication system 54.

Sensors 38 are optional and can include light sensors, biometric sensorsand other sensors known in the art that can be used to detect conditionsin the environment of system 20 and to convert this information into aform that can be used by processor 34 of system 20. Sensors 38 can alsoinclude one or more video sensors 39 that are adapted to capture images.Sensors 38 can also include biometric or other sensors for measuringinvoluntary physical and mental reactions such sensors including, butnot limited to, voice inflection, body movement, eye movement, pupildilation, body temperature, and p4000 wave sensors.

Memory 40 can include conventional memory devices including solid-state,magnetic, optical or other data-storage devices. Memory 40 can be fixedwithin system 20 or it can be removable. In the embodiment of FIG. 15,system 20 is shown having a hard drive 42, a disk drive 44 for aremovable disk such as an optical, magnetic or other disk memory (notshown) and a memory card slot 46 that holds a removable memory 48 suchas, a removable memory card, and has a removable memory interface 50 forcommunicating with removable memory 48. Data including, but not limitedto, control programs, digital images and metadata can also be stored ina remote memory system 52 such as a personal computer, computer networkor other digital system. Remote memory system 52 can also includesolid-state, magnetic, optical or other data-storage devices.

In the embodiment shown in FIG. 15, system 20 has a communication system54 that in this preferred embodiment can be used to communicate with anoptional remote memory system 52, an optional remote display 56, oroptional remote input 58. The optional remote memory system 52, optionalremote display 56, optional remote input 58 can all be part of a remotesystem 35 having the remote input 58 having remote input controls 58 c(also referred to herein as “remote input 58”), can include the remotedisplay 56, and that can communicate with communication system 54wirelessly as illustrated or can communicate in a wired fashion. In analternative embodiment, a local input station including either or bothof a local display 66 and local input controls 68 (also referred toherein as “local user input 68”) can be connected to communicationsystem 54 using a wired or wireless connection.

Communication system 54 can include for example, one or more optical,radio frequency or other transducer circuits or other systems thatconvert image and other data into a form that can be conveyed to aremote device such as remote memory system 52 or remote display 56 usingan optical signal, radio frequency signal or other form of signal.Communication system 54 can also be used to receive a digital image andother data from a host or server computer or network (not shown), aremote memory system 52 or the remote input 58. Communication system 54provides processor 34 with information and instructions from signalsreceived thereby. Typically, communication system 54 will be adapted tocommunicate with the remote memory system 52 by way of a communicationnetwork such as a conventional telecommunication or data transfernetwork such as the internet, a cellular, peer-to-peer or other form ofmobile telecommunication network, a local communication network, such asa wired or wireless local area network or any other conventional wiredor wireless data transfer system. In one useful preferred embodiment,the system 20 can provide web access services to remotely connectedcomputer systems (e.g. remote systems 35) that access the system 20through a web browser. Alternatively, remote system 35 can provide webservices to system 20 depending on the configurations of the systems.

User input system 26 provides a way for a user of system 20 to provideinstructions to processor 34. This permits such a user to make adesignation of content data files to be used in generating animage-enhanced output product and to select an output form for theoutput product. User input system 26 can also be used for a variety ofother purposes including, but not limited to, permitting a user toarrange, organize and edit content data files to be incorporated intothe image-enhanced output product, to provide information about the useror audience, to provide annotation data such as voice and text data, toidentify characters in the content data files, and to perform such otherinteractions with system 20 as will be described later.

In this regard user input system 26 can include any form of transduceror other device capable of receiving an input from a user and convertingthis input into a form that can be used by processor 34. For example,user input system 26 can include a touch screen input, a touch padinput, a 4-way switch, a 6-way switch, an 8-way switch, a stylus system,a trackball system, a joystick system, a voice recognition system, agesture recognition system a keyboard, a remote control or other suchsystems. In the preferred embodiment shown in FIG. 15, user input system26 includes an optional remote input 58 including a remote keyboard 58a, a remote mouse 58 b, and a remote control 58 c and a local input 68including a local keyboard 68 a and a local mouse 68 b.

Remote input 58 can take a variety of forms, including, but not limitedto, the remote keyboard 58 a, remote mouse 58 b or remote controlhandheld device 58 c illustrated in FIG. 15. Similarly, local input 68can take a variety of forms. In the preferred embodiment of FIG. 15,local display 66 and local user input 68 are shown directly connected toprocessor 34.

As is illustrated in FIG. 16, local user input 68 can take the form of ahome computer 36 having a processor 34 and disc storage 44, an editingstudio, or kiosk 70 (hereafter also referred to as an “editing area 70”)that can also be a remote system 35 or system 20. In this illustration,a user 72 is seated before a console including a local keyboard 68 a andmouse 68 b and a local display 66 which is capable, for example, ofdisplaying multimedia content. As is also illustrated in FIG. 16,editing area 70 can also have sensors 38 including, but not limited to,video sensors 39, digital cameras 89, audio sensors 74 and other sensorssuch as multispectral sensors that can monitor user 72 during aproduction session.

Referring back to FIG. 15, output system 28 is used for renderingimages, text or other graphical representations in a manner that permitsimage-product designs to be combined with user items and converted intoan image product. In this regard, output system 28 can include anyconventional structure, system, or output device 32 that is known forprinting or recording images, including, but not limited to, printer 29.Printer 29 can record images on a tangible surface 30 using a variety ofknown technologies including, but not limited to, conventionalfour-color offset separation printing or other contact printing, silkscreening, dry electrophotography such as is used in the NexPress 2100printer sold by Eastman Kodak Company, Rochester, N.Y., USA, thermalprinting technology, drop-on-demand inkjet technology and continuousinkjet technology. For the purpose of the following discussions, printer29 will be described as a type of printer that generates color images.However, it will be appreciated that the claimed methods and apparatusherein can be practiced with a printer 29 that prints monotone imagessuch as black and white, grayscale, or sepia toned images. As will bereadily understood by those skilled in the art, a system 35, 20 withwhich a user interacts to define a user-personalized image product canbe separated from a remote system (e.g. 35, 20) connected to a printer,so that the specification of the image product is remote from itsproduction.

In certain embodiments, the source of content data files 24, user inputsystem 26 and output system 28 can share components.

Processor 34 operates system 20 based upon signals from user inputsystem 26, sensors 38, memory 40 and communication system 54. Processor34 can include, but is not limited to, a programmable digital computer,a programmable microprocessor, a programmable logic processor, a seriesof electronic circuits, a series of electronic circuits reduced to theform of an integrated circuit, or a series of discrete components. Thesystem 20 of FIGS. 15 and 16 can be employed to make and display animage product according to a preferred embodiment of the presentinvention.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   9, 18 Number of Images-   20 system-   22 housing-   24 source of content data files-   26 user input system-   28 output system-   29 printer-   30 tangible surface-   32 output device-   34 processor-   35 remote system-   36 computer-   38 sensors-   39 video sensors-   40 memory-   42 hard drive-   44 disk drive-   46 memory card slot-   48 removable memory-   50 memory interface-   52 remote memory system-   54 communication system-   56 remote display-   58 remote input-   58 a remote keyboard-   58 b remote mouse-   58 c remote control-   66 local display-   68 local input-   68 a local keyboard-   68 b local mouse-   70 home computer, editing studio, or kiosk (“editing area”)-   72 user-   74 audio sensors-   89 digital camera-   110 data processing system-   120 peripheral system-   130 user interface system-   140 data storage system-   250 analyze images step-   255 determine identities step-   300 type count-   320 image type distribution-   500 provide image collection step-   505 provide image-type distribution step-   510 compare image types step-   511 determine match step-   515 select digital images step-   520 select image-based product step-   525 assemble image-based product step-   530 make image-based product step-   535 deliver image-based product step-   550 receive digital images step-   555 analyze digital images step-   560 determine image types step-   565 receive theme step-   570 analyze digital images step-   575 determine theme step-   580 receive image-type distribution step-   600 analyze match step-   605 communicate missing image types step-   606 receive missing image types step-   607 obtain missing image types step-   608 communicate obtained images step-   610 receive missing image types step-   705 provide image-type distributions step-   710 compare image types step-   711 compare matches step-   712 select best match step-   713 communicate best match optional step-   714 receive match approval optional step-   811 compare matches optional step-   812 order matches optional step-   813 communicate matches step-   814 receive match selection step-   900 iterated image-distribution step-   905 iterated digital image step-   910 image type comparison step-   915 recording step-   920 combine matches step-   925 calculate metric step-   930 compare metrics step-   935 select best image-type distribution step

1. A method of making an image-based product, comprising: providing animage collection having a plurality of digital images, each digitalimage having an image type; providing one or more image-typedistributions, each image-type distribution corresponding to a theme andincluding a distribution of image types related to the theme; using aprocessor to automatically compare the image types of the digital imagesin the image collection to the image types in the image-typedistribution; using the processor to automatically determine a matchbetween the image types in the image collection and the image types inthe image-type distribution; selecting a group of digital images fromthe image collection having a distribution of image types specified bythe determined matching image-type distribution; assembling the digitalimages in the selected group of images into an image-based product; andcausing the construction of the image-based product.
 2. The methodaccording to claim 1, further including: analyzing the match todetermine image types missing from the image collection present in theimage-type distribution; communicating the missing image types; andreceiving images having the missing image types.
 3. The method accordingto claim 1, further including: providing a plurality of image-typedistributions; using the processor to compare the image types of thedigital images in the image collection to the image types in each of theimage-type distributions; determining a match between the image types ofthe digital images in the image collection and the image types for eachof the image-type distributions; comparing the matches; and selectingthe best match.
 4. The method according to claim 1, further including:providing a plurality of image-type distributions; comparing the imagetypes of the digital images in the image collection to the image typesin each of the image-type distributions; determining a match between theimage types in the image collection and the image types for each of theimage-type distributions; and communicating the matches or themesassociated with the image-type distributions corresponding to thematches.
 5. The method according to claim 4, further including:comparing the matches; ordering the matches; and communicating theordered matches.
 6. The method according to claim 4, further includingreceiving a match or theme associated with an image-type distribution.7. The method according to claim 1, further including receiving one ormore image-type distributions.
 9. The method according to claim 1,further including analyzing the plurality of digital images to determineone or more themes.
 10. The method according to claim 1, furtherincluding providing one or more image-type distributions associated withan image-based product.
 11. The method according to claim 1, furtherincluding providing a plurality of image-type distributions associatedwith one theme.
 12. The method according to claim 1, wherein the imagetypes include one or more of the types: introduction type, charactertype, person type, object type, action type, and conclusion type. 13.The method according to claim 1, wherein the digital images have atemporal association and the image-type distribution includes an imagetype time order.
 14. The method according to claim 1, wherein theimage-type distribution includes a specified distribution of person orcharacter image types.
 15. The method according to claim 1, wherein animage type includes an identified person type.
 16. The method accordingto claim 1, wherein the image product is a photo-book or aphoto-collage.
 17. The method according to claim 1, further includinganalyzing the plurality of digital images to identify themes associatedwith the plurality of digital images.
 18. The method according to claim1, further including causing the construction of an electronicimage-based product or a printed image-based product.